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Improving vocabulary and reading comprehension is critical 

to students’ academic success, particularly for English learners 
who may have less exposure to academic language than native 
English-speaking students.' Word Generation is a supplemental 
program that aims to improve students’ reading comprehen- 
sion by building students’ vocabulary, academic language, 

and perspective-taking skills through classroom discussion 

and debate. Word Generation was designed for all students; 
however, English learners in particular could benefit from its 
focus on academic language. Word Generation consists of a 
series of interdisciplinary units with daily lessons focused on a 
high-interest topic to increase student engagement. Each unit 
targets a small number of academic vocabulary words that 

are integrated into texts, activities, writing tasks, debates, and 
discussions across content areas. Several Word Generation pro- 
grams exist. In the Word Generation Weekly (WordGen Weekly) 
and Word Generation Elementary (WordGen Elementary) pro- 
grams, units are designed to be used across English language 


arts, math, science, and social studies in grades 6—8 and grades 
4 and S, respectively. In the Science Generation (SciGen) and 
Social Studies Generation (SoGen) programs, units can supple- 
ment or be used in place of regular science and social studies 
curriculum units in grades 6-8. The different Word Generation 
programs can be implemented separately or together. 


This What Works Clearinghouse (WWC) report, part of the 
WWC’s English Learners topic area, explores the effects of 
Word Generation on English language proficiency and reading 
comprehension for English learners. The WWC identified nine 
studies of Word Generation that focused on English learners. 
One of these studies meets WWC standards. The evidence 
presented in this report is from one study of the impact of 
Word Generation on English learners in a range of grades— 
4th through 7th—and in a variety of school settings, including 
elementary, middle, and K—8 schools in urban, suburban, and 
small town districts. 


What Happens When Students Participate in Word Generation?? 


The evidence indicates that implementing Word Generation: 
¢ May result in little or no change in English language 
proficiency. 


¢ May result in little or no change in reading 
comprehension. 


Findings on Word Generation from the one study that meets 
WWC standards are shown in Table |. The table reports an 
effectiveness rating, the improvement index, and the number 
of studies and students that contributed to the findings. The 
improvement index is a measure of the intervention’s effect 
on an outcome. It can be interpreted as the expected change 
in percentile rank for an average comparison group student if 
that student had received the intervention. 


Table 1. Summary of findings on Word Generation from the one study that meets WWC standards 


Study Findings Evidence meeting WWC standards (version 4.0) 
Improvement index 
Outcome domain Effectiveness rating (percentile points) Number of studies Number of students 
English language proficiency No discernible effects +9 1 241 
Reading comprehension No discernible effects +4 1 241 


Note: The improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention. 
For example, an improvement index of +9 means that the expected percentile rank of the average comparison group student would increase by 9 points if the student received 
Word Generation. The improvement index values are generated based on one finding from the outcome analysis that meets WWC standards conducted within each domain, as 
reported by Kim et al. (2018). A positive improvement index does not necessarily mean the estimated effect is statistically significant. The English language proficiency outcome 
reported in the study is the Core Academic Language Skills-Instruments (CALS-I). The reading comprehension outcome reported in the study is the Global Integrated Scenario- 
based Assessment (GISA). The effects of Word Generation are not known for other outcomes within the English Learners topic area, including alphabetics, reading fluency, 
general literacy achievement, writing conventions, writing productivity, writing quality, mathematics achievement, science achievement, and social studies achievement. 


BOX 1. HOW THE WWC REVIEWS AND DESCRIBES EVIDENCE 


The WWC evaluates evidence based on the quality and results of reviewed studies. The criteria the WWC uses for evaluating 
evidence are defined in the Procedures and Standards Handbooks and the Review Protocols. The studies summarized in this report 
were reviewed under WWC Standards (version 4.0) and the Primary Mathematics topic area protocol (version 4.0). 


To determine the effectiveness rating, the WWC considers what methods each study used, the direction of the effects, and the 
number of studies that tested the intervention. The higher the effectiveness rating, the more certain the WWC is about the reported 
results and about what will happen if the same intervention is implemented again. The following key explains the relationship between 


effectiveness ratings and the statements used in this report: 


Effectiveness Rating Rating interpretation 


Description of the evidence 


Positive (or negative) effects The intervention is likely to change an Strong evidence of a positive effect, with no 


outcome 


overriding contrary evidence 


Potentially positive (or negative) effects The intervention may change an outcome Evidence of a positive effect with no overriding 


contrary evidence 


No discernible effects The intervention may result in little to no No affirmative evidence of effects 


change in an outcome 


Mixed effects The intervention has inconsistent effects Evidence includes studies in at least two of 


onan outcome 


How is Word Generation Implemented? 


The following section provides details of how Word Gener- 
ation was implemented. This information can help educators 
identify the requirements for implementing Word Generation 
and determine whether implementing this intervention would 
be feasible in their district or school. Information on Word 
Generation presented in this section comes from the study 
that meets WWC standards (Kim et al., 2018) and from cor- 
respondence with the developer. 


¢ Goal: Word Generation aims to improve students’ reading 
comprehension by building students’ vocabulary, academic 
language, and perspective-taking skills through classroom 
discussion and debate. 


¢ Target population: Word Generation can be used for 
students in grades 4 to 8. WordGen Weekly, SciGen, and 
SoGen are used with students in grades 6-8. WordGen 
Elementary is used with students in grades 4 and 5. 


¢ Method of delivery: WordGen Weekly and WordGen 
Elementary are supplemental interdisciplinary units that 
are integrated into English language arts, science, social 
studies, and mathematics instruction. SciGen and SoGen 
units can be used in place of the regular curriculum or as 
a supplement. Each unit includes additional activities for 
other content areas to provide cross-disciplinary reinforce- 
ment of academic vocabulary. 


¢ Frequency and duration of service: WordGen Weekly 
consists of three series of 24 weekly units, with each series 
designed to be used over one academic year. Series and 
units within series can be used in any order. Each unit 
within the series includes five target vocabulary words and 


these categories: studies with positive effects, 
studies with negative effects, or more studies 
with indeterminate effects than with positive or 
negative effects 


Comparison group: In the study that contributes to 
this intervention report, students in the comparison 
group were taught using a wide range of instructional 


practices and standard programs that sought to improve 
vocabulary. These programs were highly variable 
across schools in the comparison group. 


five connected activities that typically last from 15 to 20 
minutes and take place during regular English language 
arts, science, social studies, and math instruction. 


WordGen Elementary consists of an introductory unit (one 
week) and 12 additional units (two weeks each) for each 
grade level (4 and 5). Each two-week unit includes daily 
40- to 50-minute lessons. Schools can choose to implement 
some or all of the units across the school year. 


SciGen and SoGen each offer six discipline-specific units 
for each grade level (6-8). Each unit is one week in length 
and includes daily 40- to 50-minute lessons in science 

or social studies plus 15- to 20-minute supplementary 
activities that can be integrated into English language arts, 
mathematics and science (for SoGen) or social studies (for 
SciGen) instruction. Schools can choose to implement 
some or all of the units across the school year. 


Intervention components: Word Generation includes 
several key components and features, as described in 
Table 2. The four different Word Generation programs can 
be implemented separately or together. 


Table 2. Components of Word Generation 


Core component Description 


Word Generation units | Each Word Generation unit is organized around a high-interest topic that aims to engage students’ interest—for example, 
“Should the school day be lengthened?” or “Should the government impose a mandatory year of service after high school?” 
Students read or listen to multiple points of view on a topic at the beginning of the week, learn target vocabulary words 
relevant to that topic, prepare and debate the topic, and finally write an argumentative piece on the topic, all using the 
target vocabulary words. Instruction prioritizes student participation and teamwork to improve other important skills such as 


academic language and perspective-taking. 


Cross-discipline Each Word Generation unit includes complementary activities across disciplines (English language arts, math, science, and 
activities social studies) to provide students with multiple opportunities to reinforce vocabulary learning and to use academic language 


in a variety of academic contexts. 


What Does Word Generation Cost? 


This preliminary list of costs is not designed to be exhaus- 
tive; rather, it provides educators an overview of the major 
resources needed to implement Word Generation. The 
program costs described below are based on the information 
available as of April 2019. 


¢ Equipment and materials costs: The student materials 
and teacher resources for implementing Word Generation 
can be downloaded for free and then printed and distrib- 
uted for noncommercial purposes from the program’s web- 
site (wordgen.serpmedia.org). As an option, students can 
access digital materials for Word Generation on tablets. 
There is no cost to download these materials or to access 
videos and other resources for teachers on vocabulary 
development and classroom discourse. School districts can 
purchase printed copies from a print-on-demand company 
linked to the program’s website. 


Personnel costs: Word Generation can be adopted without 
professional learning support from the developer, Strategic 
Education Research Partnership (SERP). However, SERP 
does offer professional development upon request to improve 
implementation fidelity. Trainings range from one to three 
days in duration. Costs range from $3,000 to $12,000 
(excluding travel and materials). 


Facilities costs: The program is delivered in a classroom 
setting during regular class time. 


Costs paid by students or parents: None. 
In-kind supports: No information is available. 


Sources of funding: School districts or schools usually 
purchase materials and/or professional development for 
Word Generation. 


For More Information: 
About Word Generation 


Strategic Education Research Partnership, 1100 Connecticut Ave NW, Suite 1310, Washington, DC 20036 


Attn: Suzanne Donovan 


Email: info@serpinstitute.org. Web: https://wordgen.serpmedia.org/. Phone: (202) 223-8555 


About the cost of the intervention 
Web: https://wordgen.serpmedia.org/ 


Research Summary 


The WWC identified nine studies that investigated the effec- 
tiveness of Word Generation for English learners (Figure 1): 


¢ One study meets WWC group design standards without 
reservations. 


¢ Four studies do not meet WWC group design standards. 
¢ Four studies are ineligible for review. 


The WWC reviews findings on the intervention’s effects for 
English learners on eligible outcome domains from studies 
that meet standards, either with or without reservations. 
Based on this review, the WWC generates an effectiveness 
rating, which summarizes how the intervention impacts, or 
changes, a particular outcome domain. The WWC reports 
additional supplemental findings, such as those the study 


Figure 1. Effectiveness ratings for Word Generation 
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authors reported on non-English learners, on the WWC 
website (https://whatworks.ed.gov). These supplemental find- 
ings and findings from studies that either do not meet WWC 
standards or are ineligible for review do not contribute to the 
effectiveness ratings. 


The one study of Word Generation that meets WWC group 
design standards reported findings on English language pro- 
ficiency and reading comprehension. No other findings in the 
study met WWC group design standards within any outcome 
domain included in the English Learners topic area.* Cita- 
tions for the nine studies reviewed for this report are listed in 
the References section, which begins on page 9. Citations for 
the four studies that are ineligible for review and the reasons 
the WWC determined they were ineligible are also listed in 
the References section. 
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Word Generation has no discernible effects on English language proficiency 


One study that meets WWC group design standards without reservations contributes findings to the English language 
proficiency domain. The WWC determined one study showed evidence of an indeterminate effect of Word Generation 


on English language proficiency (Kim et al., 2018). 


Word Generation has no discernible effects on reading comprehension 


One study that meets WWC group design standards without reservations contributes findings to the reading 
comprehension domain. The WWC determined one study showed evidence of an indeterminate effect of Word Generation 


on reading comprehension (Kim et al., 2018). 


Main Findings 


Table 3 shows the findings from the one Word Generation WWC standards, the effectiveness rating for English lan- 
study that meets WWC standards. The table includes WWC guage proficiency is no discernible effects. These findings 
calculations of the mean difference, effect size, and perfor- are based on 241 English learners. The effectiveness rating 
mance of the intervention group relative to the comparison for reading comprehension is no discernible effects. These 
group. Based on findings from the one study that meets findings are based on 241 English learners. 


Table 3. Findings by outcome domain from the one study of Word Generation that meets WWC Standards 


Mean 
(standard deviation) WWC calculations 
Sample Intervention Comparison Mean Effect Improvement 
Measure (study) Study sample size group group difference __ size index p-value 
Core Academic 4th- to 7th-grade 241 0.11 -0.07 0.18 0.22 +9 03 
Language Skills- students (0.73) (0.87) 
Instruments (CALS-I; 
Kim et al., 2018)? 
Outcome average for English language proficiency from one study 0.22 9 
Global Integrated 4th- to 7th-grade 241 938.94 937.7 1.24 0.02 +4 95 
Scenario-based students (49.38) (56.20) 
Assessments (GISA; 
Kim et al., 2018) 
Outcome average for reading comprehension from one study 0.02 1 


Notes: For mean difference and effect size values reported in the table, a positive number favors the intervention group and a negative number favors the comparison group. 
The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are given the 
intervention (measured in standard deviations of the outcome measure). An indicator of the effect of the intervention, the improvement index can be interpreted as the expected 
change in percentile rank for an average comparison group student if that student had received the intervention. For example, an improvement index of +9 means that the 
expected percentile rank of the average comparison group student would increase by 9 points if the student received Word Generation. A positive improvement index does not 
necessarily mean the estimated effect is statistically significant. Some statistics may not sum as expected due to rounding. 

a Kim et. al. (2018) did not require any corrections for multiple comparisons nor difference-in-differences adjustments. The p-values presented here were calculated by the 
WWC, because the p-values reported in the study did not account for the clustered assignment of students to conditions by school. This report presents the unadjusted means 
provided in the original study because the adjusted means came from models that used an endogenous covariate. The study is characterized as having indeterminate effects 
on English language proficiency and reading comprehension because the mean effect reported is not statistically significant. For more information, please refer to the WWC 
Procedures Handbook, version 4.0, page 22. 


In What Context Was Word Generation Studied? 


The following section provides information on the setting of the can help educators understand the context in which the study 
one study of Word Generation that meets WWC standards, and of Word Generation was conducted, and determine whether the 
a description of the participants in the research. This information program might be suitable for their setting. 


WHERE THE STUDY WAS CONDUCTED 


1 study, 241 students in 3 districts the northeastern United States. 


In urban, suburban, and small town districts 


English Learners: 100% 


caus * K 4 2 7 OOOO 


Grades 4—7 


Details of Each Study that Meets WWC Standards 


This section presents details for the one study of Word in experimental effects of Word Generation in high-poverty 
Generation that meets WWC standards. These details include schools. International Journal of Bilingual Education and 
the full study reference, findings description, findings sum- Bilingualism. 


mary, and description of study characteristics. A summary of 
domain findings for the study is presented below, followed 
by a description of the study characteristics. These study- 
level details include contextual information around the study 
setting, methods, sample, intervention group, comparison 
group, outcomes, and implementation details. For additional 
information, readers should refer to the original study. 


Findings from Kim et al. (2018) show evidence of an inde- 
terminate effect of Word Generation in the English language 
proficiency and reading comprehension domains (Table 4). 
These findings are based on one outcome measure in each 
domain, each including 241 English learners. Two citations 
contributed findings and context to the review of Kim et al. 
(2018), including the primary citation above. The full list of 
Research details for Kim et al. (2018) citations reviewed for this report is in the References section, 


which begins on page 9. 
Kim H. Y., Hsin, L. B., Snow C. E. (2018). Reducing aca- 


demic inequalities for English language learners: variation 


Table 4. Summary of findings from Kim et al. (2018) 


Meets WWC Group Design Standards Without Reservations 


Study findings 
Sample Average Improvement Statistically 
Outcome domain size effect size index significant 
English language proficiency 8 schools, 241 students +0.22 9 No 
Reading comprehension 8 schools, 241 students +0.02 1 No 


Table 5. Description of study characteristics for Kim et al. (2018) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 
group 


Meets WWC Group Design Standards Without Reservations. This is a cluster randomized controlled trial (RCT) with low 
cluster-level attrition and individual-level nonresponse. For more information on how the WWC assigns study ratings, please 
see the WWC Procedures and Standards Handbooks (version 4.0) and WWC Standards Briefs, available on the WWC website. 


The study sample included nine K-8 and one elementary-middle school pair, which was considered to be a single school. 
Of the 10 schools, the analysis that meets WWC standards included 8. The 10 schools were in three school districts located 
in the northeast region of the United States. One district was located in a city, another in a suburban community, and the 
last in a small town. Nine of the schools served students in grades K-8, and the other one was an elementary—middle 
school pair, in which the elementary school was a feeder school for the middle school. 


The study used a cluster-level RCT design. The researchers created five pairs of schools from within the same district that 
had similar characteristics, including size, percentage of language-minority students, percentage of students receiving free 
and reduced-price lunch, and student proficiency on state tests. The study regarded the feeder elementary school and 
middle school to be a single school. The schools in each matched pair were then randomly assigned to an intervention or 
comparison group. The analytic sample includes students who entered schools after random assignment, but this review 
did not consider these joiners to pose a risk of bias, as it is unlikely that a school’s assigned condition influenced whether 
students enrolled in the school. Word Generation is a supplementary program, and likely to be unfamiliar to parents at the 
time the study was conducted. 


At follow-up, the eight schools in the analytic sample included 310 English learners, 138 in the intervention group and 
172 in the comparison group. 


For both outcome measures, the sample loss of schools after random assignment (attrition) was within the acceptable 
threshold for the review: overall attrition was 20%, and differential attrition was 0 percentage points. The nonresponse of 
English learners within the remaining schools at follow-up was also within the acceptable threshold for the review: overall 
nonresponse was 22% and differential nonresponse was 7 percentage points. 


The authors described two of the three districts in the study as serving an ethnically diverse and low-income population and 
the third district as serving a primarily white and low- to middle-income population. The sample includes students in grades 
4-7. The schools in the study included between 11% to 14% English learners. The 241 students that contributed outcomes 
to the findings in Table 4 were all English learners. 


Students in this study received the following Word Generation programs. Students in 4th and 5th grades received WordGen 
Elementary (12 two-week units with daily 40- to 50-minute lessons). Students in 6th and7th grades received SciGen and 
SoGen (six one-week units with daily 45-minute lessons for each SciGen and SoGen). Students in 6th and 7th grade 

also received 12 one-week units with daily 20-minute lessons of WordGen Weekly. This implementation differs from the 
developer's current description of WordGen Weekly, which suggests using 24 units across a school year. 


Students in grades 5-7 when outcomes were measured who were present in the same school during the previous school 
year received two years of the intervention (63% of students received two years of the intervention). Students in grades 5-7 
who enrolled in the schools only during the year when outcomes were measured and all grade 4 students received just one 
year of the intervention (37% of students received one year of the intervention). 


Comparison Students in the comparison schools received instruction to improve vocabulary development through a wide range of 


group programs and strategies, but the study did not collect detailed information on them. 
Outcomes and Study authors reported findings on two outcome measures that are eligible for review under the English Learners topic area: 
measurement the Core Academic Language Skills Instruments (CALS-I) and the Global Integrated Scenario-based Assessments (GISA). 


These outcome measures were reviewed under the English language proficiency and reading comprehension domains, 
respectively. Both outcomes are measured in the spring of 2014 after the intervention had been offered in the intervention 
schools for nearly two school years. 


The study also reports findings separately for non-English learners on two eligible outcomes, CALS-I and GISA, 
for the first and second years of implementation. These supplemental findings are reported on the WWC website 
(httos://whatworks.ed.gov). The supplemental findings do not factor into the intervention’s rating of effectiveness. 


The study also reports findings for English learner students on the CALS-| and GISA outcomes for the first year of 
implementation. These findings do not meet WWC group design standards because they are based on a cluster RCT 
with high levels of overall and differential nonresponse of students and the students in the schools at follow-up are not 
representative of the clusters. In addition, the study reports findings for English learner students on a Word Generation 
vocabulary measure for the first and second years of implementation. These findings do not meet WWC group design 
standards because the WWC determined this measure to be overaligned with the intervention. The study also reports 
findings on students’ social perspective articulation and positioning. These findings were not eligible for review under the 
English Learners protocol (version 4.0). 


Additional Teachers in the intervention group were invited to a three-day summer institute at which they learned about the Word 
implementation Generation curriculum and underlying principles. Not all teachers could attend the summer training, but no information is 
details provided on what percentage attended. The training was repeated before the start of Year 2. Word Generation coaches 


supported implementation by responding to teacher queries, modelling lessons, and organizing school learning groups. 
However, the study reported that teachers received varying amounts of coaching. 
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'Snow, C. E., & Uccelli, P. (2009). The challenge of academic 
language. In D. R. Olson & N. Torrance (Eds.), The Cambridge 
handbook of literacy (pp. 112-133). Cambridge: Cambridge 
University Press. https://www.cambridge.org/core/books/cam- 
bridge-handbook-of-literacy 


>The descriptive information for this intervention comes from the 
overview on the developer’s website (https://wordgen.serpmedia. 
org/overview.html). The WWC requests developers review the 
intervention description sections for accuracy from their perspec- 
tive. The WWC provided the developer with the intervention 
description in April 2019 and the WWC incorporated feedback 
from the developer. Further verification of the accuracy of the 
descriptive information for this intervention is beyond the scope of 
this review. 


3 The literature search reflects documents publicly available by Feb- 
ruary 2020. Reviews of the studies in this report used the standards 
from the WWC Procedures and Standards Handbook (version 4.0) 
and the English Learners review protocol (version 4.0). The evidence 
presented in this report is based on available research. Findings and 
conclusions could change as new research becomes available. 

‘The effects of Word Generation are not known for other out- 
come domains within the English Learners topic area, including 
alphabetics, reading fluency, general literacy achievement, writing 
conventions, writing productivity, writing quality, mathematics 
achievement, science achievement, and social studies achievement. 
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